Search for: All records

Foundation models (FM) have shown immense human-like capabilities for generating digital media. However, foundation models that can freely sense, interact, and actuate the physical domain is far from being realized. This is due to 1) requiring dense deployments of sensors to fully cover and analyze large spaces, while 2) events often being localized to small areas, making it difficult for FMs to pinpoint relevant areas of interest relevant to the current task. We propose FlexiFly, a platform that enables FMs to “zoom in” and analyze relevant areas with higher granularity to better understand the physical environment and carry out tasks. FlexiFly accomplishes by introducing 1) a novel image segmentation technique that aids in identifying relevant locations and 2) a modular and reconfigurable sensing and actuation drone platform that FMs can actuate to “zoom in” with relevant sensors and actuators. We demonstrate through real smart home deployments that FlexiFly enables FMs and LLMs to complete diverse tasks up to 85% more successfully. FlexiFly is critical step towards FMs and LLMs that can naturally interface with the physical world. 

We present the design and implementation of RECA, a novel human-centric recommender system for co-optimizing energy consumption, comfort and air quality in commercial buildings. Existing works generally optimize these objectives separately, or by only controlling energy consuming resources within the building without directly engaging occupants. We develop a deep reinforcement learning architecture based on multitask learning, demonstrate how it can be used to jointly learn energy savings, comfort and air quality improvements for different actions, and build a recommender system with humans-in-the-loop. Through real deployments in multiple commercial buildings, we found that RECA has the potential to further reduce energy consumption by up to in energy-focused optimization, improve all objectives by in joint optimization, and improve thermal comfort by up to in comfort and air quality focused optimization, over existing solutions. 

Existing large language models (LLMs) that mainly focus on Standard American English (SAE) often lead to significantly worse performance when being applied to other English dialects. While existing mitigations tackle discrepancies for individual target dialects, they assume access to high-accuracy dialect identification systems. The boundaries between dialects are inherently flexible, making it difficult to categorize language into discrete predefined categories. In this paper, we propose DADA (Dialect Adaptation via Dynamic Aggregation), a modular approach to imbue SAE-trained models with multi-dialectal robustness by composing adapters which handle specific linguistic features. The compositional architecture of DADA allows for both targeted adaptation to specific dialect variants and simultaneous adaptation to various dialects. We show that DADA is effective for both single task and instruction finetuned language models, offering an extensible and interpretable framework for adapting existing LLMs to different English dialects. 

With the global spread of the COVID-19 pandemic, ventilation indoors is becoming increasingly important in preventing the spread of airborne viruses. However, while sensors exist to measure wind speed and airflow gradients, they must be manually held by a human or an autonomous vehicle, robot, or drone that moves around the space to build an airflow map of the environment. In this demonstration, we present DAE, a novel drone-based system that can automatically navigate and estimate air flow in a space without the need of additional sensors attached onto the drone. DAE directly utilizes the flight controller data that all drones use to self-stabilize in the air to estimate airflow. DAE estimates airflow gradients in a room based on how the flight controller adjusts the motors on the drone to compensate external perturbations and air currents, without the need for attaching additional wind or airflow sensors. 

There has been an immense growth in sensors, actuators, and smart devices in recent years, which enable us to better sense, actuate, and understand the physical world. Despite this growth, we have yet to achieve fully intelligent environments. This is, in part, due to the large number of different organizations creating smart devices with proprietary technologies and communication protocols that are not compatible with each other and require significant engineering to incorporate and adapt to specific applications. In this work, we present an easy-to-install and low-cost embedded platform that allows users to rapidly configure a mixture of sensors and actuators. The system is based on the commonly-used Raspberry Pi ecosystem, easily configurable, and does not require users to have prior knowledge of programming, which allows anyone, regardless of background, to use. We also introduce a battery-powered wireless extension module that is suitable for mobile drone applications, where a chord-powered Raspberry Pi is not suitable. We demonstrate the impact our system has on enabling drones with flexible sensing modalities and creating smarter environments by integrating our platform into a variety of intelligent home applications.